Deep Convolution Neural Networks, or Deep CNN is the most core of the remarkable development in the field of Deep Learning. Though the CNN has been employed to solve character recognition problems in 1990s, it is not until recently that the CNN has become widespread in Machine Learning. For example, in 2012, the CNN significantly outperformed its competitors in an annual software contest, the ImageNet Large Scale Visual Recognition Challenge, and won the contest. After that, the CNN has become a very useful tool in the field of the machine learning.
However, there was a prejudicial perception that 32-bit floating point operations are needed for the deep learning algorithms, so that mobile devices may be considered as not being capable of performing programs including the deep learning algorithms.
However, by some experiments, it was proved that 10-bit fixed point operations, which require less computing resources than the 32-bit floating point operations, are sufficient for the deep learning algorithms. Thus, there were many attempts to apply the 10-bit fixed point operations to the deep learning algorithms by using devices with limited resources, i.e., the mobile devices.
There were many successful methods for quantizing values in a 32-bit floating point form to values in a 10-bit fixed point form. But, those had a critical shortcoming. On condition that values included in a plurality of channels vary a lot, values of one or more specific channels, including small values, among the channels may be ignored. It can be seen in FIG. 5.
FIG. 5 shows example values of the different channels whose values vary a lot.
By referring to FIG. 5, it can be seen that values included in a first channel of a first feature map, outputted from a first convolutional block 210-1 are (0.64, 0.65, 0.63), and values included in a second channel thereof are (0.002, 0.001, 0.0019). If the values of the first channel and those of the second channel are quantized with conventional techniques, a unit value to be used for the quantization is determined according to the first channel or the second channel.
In case the unit value is determined according to the first channel, the unit value gets larger to represent the values included in the first channel. Then, the values included in the second channel can be quantized as 0, because the unit value is too big for the values included in the second channel. On the contrary, in case the unit value is determined according to the second channel, the unit value gets smaller to represent the values included in the second channel. Then, the values included in the first channel cannot be quantized properly, because the unit value is too small.
If values of a specific channel are ignored or if values of the specific channel are not quantized properly as shown above, an output of the CNN may be distorted.